This invention relates to apparatus and methods for controlling the scoring or grading of nonobjective assessment materials (such as essays handwritten on paper), using scorers or readers at diverse geographic locations. More particularly, the invention relates to apparatus and methods for allocating appropriate activities to scorers, distributing materials to scorers required for those activities, monitoring scorers"" performance in those activities, and recording the results of the scorers"" activities. The present invention provides improved control methods for scoring xe2x80x9con screenxe2x80x9d, a technique marketed under the trademark xe2x80x9cScoreImagexe2x80x9d. In general, the ScoreImage technique is described in the commonly assigned U.S. Pat. No. 5,672,060 (the disclosure of which is hereby incorporated by reference).
In on-screen scoring, nonobjective assessment materials are scored through the use of digitized images depicting the responses made for the assessment. Such digitized images are displayed on computer screens for scoring or other judgments of the assessment materials.
Through the use of captured images as the representation of the assessment materials, the need to manage and control large amounts of paper in a secure manner is eliminated. The assessment materials themselves can be kept secure at a scanning center, and need not be handled again for any part of the scoring process. Moreover, since no names or other identifying information are sent to scorers with the images of the materials to be scored, student anonymity is assured and a potential source of scorer bias is removed. Each scorer may work on a personal computer or terminal, while work for the scorer can be allocated to that scorer by an independent process.
When a scorer is ready to start or continue a session, the scorer can select from among three types of activities: training, qualifying, or scoring. For each activity, the system provides a particular xe2x80x9cbatchxe2x80x9d or set of assessment items appropriate for that activity.
If supervisory personnel have assigned a scorer to a particular assessment, the scorer can select training and request or select a batch of assessment items appropriate for training or to help the scorer become proficient in assigning correct scores for items in that assessment. These so-called xe2x80x9ctraining batchesxe2x80x9d consist exclusively of pre-scored items and may provide feedback to the scorer immediately after the scorer enters his or her score. Training batches can be preestablished or designated by supervisory personnel, and a copy of the batch can be made for the scorer, although, alternatively, training batches can be dynamically built upon request. The scorer""s performance is recorded in appropriate detail and summary files so that supervisory personnel, that is, project leaders or managers, can identify scorers who are not correctly assessing the training items.
Once a scorer has completed the appropriate set of training activities, the scorer can request a so-called xe2x80x9cqualifying batchxe2x80x9d of assessment items, so that he or she can demonstrate his or her ability to properly score the particular assessment. xe2x80x9cQualifying batchesxe2x80x9d consist exclusively of pre-scored items and are used to ensure that the scorer has properly internalized the training so that the scorer provides accurate scores to assessment items. In accordance with the invention, qualifying batches are pre-established by supervisory personnel and a copy of the batch is made for the scorer, although qualifying batches may also be dynamically built upon request. When scoring qualifying batches, unlike training batches, the scorer is typically not informed of his or her performance until after scoring of the batch is completed. After scoring of the batch is completed, the scores entered by the scorer are compared to the predetermined scores, and the scorer""s overall performance is compared to a predetermined standard. The scorer""s performance is recorded in appropriate detail and summary files and is made available or reported to project leaders or managers. It may then also be reported back to the scorer. After a scorer completes a required set of one or more qualifying batches, if the scorer""s overall performance meets or exceeds the pre-determined standard, the scorer is deemed xe2x80x9cqualifiedxe2x80x9d to obtain batches of xe2x80x9clive,xe2x80x9d or unscored assessments to score. On the other hand, if the scorer fails to meet the predetermined standard, the scorer may be required to perform additional training and take additional qualifying batches or be precluded from further participation. While most projects require that a scorer successfully meet the qualifying requirements once in any given assessment, other projects may require that the scorer take at least one qualifying batch and re-qualify each day or each scoring session.
When a scorer has become qualified, the scorer can request a batch of xe2x80x9clive,xe2x80x9d or unscored, assessment items. Either the entire set or a subset of the assessment items in a so-called xe2x80x9cAssessment Scoring Filexe2x80x9d is examined to find a subset of items which can be assigned to the scorer. Typically, live items to be scored by the reader are randomly chosen from the resulting subset of items. Often, a fixed number or percentage of pre-scored so called xe2x80x9cmonitoringxe2x80x9d items are also selected and are combined with the live assessment items to form a complete xe2x80x9cbatchxe2x80x9d of items to be scored. The scorer then scores each of the assessment items in the batch as described in the above-mentioned application, and the resulting scores or judgments and other information are stored in a xe2x80x9cScorer Batch Filexe2x80x9d. As soon as the scorer completes scoring of a batch, the scored results are processed into the system. Scores given to live assessment items are stored in the appropriate assessment records, and scores given to prescored, or monitoring, items are compared to the predetermined correct score so that the scorer""s accuracy and precision can be monitored.
When the scorer completes work on any batch, the system creates several records that become part of the permanent project records. These records permit accounting for each scorer""s activities, as well as permitting evaluation of scorers.
Remote Scoring
In the above-mentioned patent, procedures were disclosed involving the sending of batches to scorers at remove sites via diskettes, with the results of scoring communicated back to the central site via regular telephone lines and computer modems.
(a) Distributing Materials to be Scored
For most batches, the files containing the images for the assessments to be scored will not fit on a single diskette, so that on-screen scoring at remote sites based on sending to scorers diskettes of materials to be scored is somewhat difficult to control and not entirely satisfactory.
The present embodiments of the invention, therefore, provide apparatus and methods to ensure that the images and other files needed for scoring are readily available to scorers when the work is started. This aspect of the invention, then, relates to the distribution of materials to scorers for scorers in a variety of settings and for projects with a variety of special needs, with emphasis on the use of scorers at diverse locations.
(b) Controlling Materials and Accountability of Scorers
Another aspect of the present invention is in the area of controlling materials and providing techniques of holding scorers accountable for their activities. The time delays involved in sending out diskettes containing batches to be scored can sometimes hamper control and management of a scoring project. In order to provide control of the materials and oversight of the scorers and the materials, it is best to allocate work to a scorer immediately before the scorer begins that work, and important that the results of that work be processed and evaluated before additional work is allocated to the scorer, so that the scorer is held accountable for his or her activity. This step of evaluation before allocating additional work is important to avoid allocating additional work to a scorer who has not given correct scores to the items he or she has scored. The process of evaluation also permits supervisory personnel to timely identify scorers who need additional training and requalification, or scorers who should be removed from the project.
The present invention therefore provides as one of its aspects, apparatus and methods to allocate to each scorer items to be scored immediately before the scorer begins working on those items, and thereafter, only after processing and evaluating all work already completed by the scorer. In addition, the present invention provides apparatus and methods to collect information for proper management and control of the scoring process, and provides apparatus and methods to ensure proper security of all information used within a scoring project.
(c) Settings for Scorers
It has heretofore been suggested that the invention be implemented by an arrangement in which computers are connected together in a network and an alternative arrangement in which scorers use a series of independent (non-interconnected) computers, generally at diverse locations. As is described below, the present invention contemplates and can exploit to advantage several different settings for scorers. Such settings include both physical proximity to a central site and particular techniques for distribution of materials and control of the open ended scoring process.
(i) Network Scoring
The preferred embodiment of the above-mentioned patent was an apparatus and method based on a series of independent (non-interconnected) computers. An alternative embodiment utilized a local area network (xe2x80x9cLANxe2x80x9d) in which all computers used by scorers are connected to a common LAN. In this so-called xe2x80x9cNetwork Scoringxe2x80x9d setting, the items to be scored are stored in a Scorer Batch File which is created as a file on the network. Within the Scorer Batch File, each record points directly to one or more images in the image database (the xe2x80x9cImageBasexe2x80x9d) so there is no need to create a special file of the images to be scored in that batch. During scoring, then, the images are drawn directly from the ImageBase of assessment item images, and the scored results or judgments are stored directly into the Scorer Batch File. Security issues are addressed in the Network Scoring setting by providing workstations that can only be used by scorers for training, qualifying, and scoring functions. By providing only xe2x80x9cdisklessxe2x80x9d computers that automatically connect to the network, scorers are blocked from making any copies of the programs or assessment materials, and carefully designed login scripts and menus restrict scorers to access only to specific programs and files.
(ii) Remote Scoring in Groups
For groups, or clusters, of remote scorers, a Remote Network Scoringxe2x80x9d setting can be used. This process is based on a semi-autonomous remote LAN for providing scoring. In this setting some or all of the ImageBase files and a copy of the Assessment Control File, are transferred from the primary scoring site to the remote network. Those skilled in the art will recognize that this transfer can be accomplished by high-speed data lines, magnetic tape, or other high volume storage devices. Once these files have been loaded in the remote LAN, the remote LAN and a central LAN at the primary scoring site (hereafter referred to as the xe2x80x9ccentral site LANxe2x80x9d) are connected via a telephone link through a control program running as a node on the central site LAN which has access to files at the remote LAN. When processing, scorers at the remote network site have exactly the same interaction as scorers at a central site, but are not connected directly to the central site LAN. Requests from scorers are forwarded over telephone lines to the central site, batches are constructed at the central site, and the batches are stored directly on the remote LAN. From then on, all scoring functions are performed as they would be on a central Network Scoring system, and the images are drawn directly from the copies of the ImageBase files which have been copied to the remote network""s file system.
In this setting, then, all file transfers are controlled and managed by computers at the central site LAN, and remote personnel are not given access to management files unless explicitly sent from the central site LAN to the remote LAN by project supervisory personnel.
(iii) Individual Remote Scoring
The present invention involves three distinct settings for individual remote scoring, each based on its own apparatus and method, and each appropriate for specific project needs. The first two are especially useful and desirable for projects requiring rapid responses and quick turnaround. The third is ideally suited for projects without such extreme time constraints but for which large numbers of assessments are prepared for scoring at one time.
[a] Individual Remote Scoring with Rapid Response
Some scoring projects require extremely rapid response, a need that cannot be met by diskettes or any physical delivery to scorers. Moreover, the bandwidth of standard analog telephone is often insufficient to transmit images quickly enough for viable scoring, so that file transfer by modem in lieu of diskettes is not a viable option. Projects requiring extremely rapid response might include applications in which scores are needed the same day as when the applicant is tested. The present invention therefore provides, in one of its aspects, apparatus and methods for two alternative settings whereby a centrally located computer system provides the distribution of materials to be scored to readers in a rapid fashion, so that scores can be quickly calculated with control over the distribution processing and security of the scoring process.
Fast Communications
One aspect of this invention, then, is based on the utilization of new technologies for the transfer of images for rapid response scoring projects. In this setting, new high-speed digital data lines (ISDN), highspeed connections to the Internet, or other technologies (e.g. cable systems) are used to transfer data in lieu of traditional dial-up telephone lines or diskettes.
Text in Lieu of Images
A second aspect of this invention capable of rapid scoring is based on the compressed character strings of text in lieu of bit-mapped images to transmit open ended assessments involving text materials. The assessment can either be created as text by the respondent typing the response directly into a computer, or the text can be derived from a captured image and converted to text by key-entry based on the image of the response at the central site as soon as the image of the written response is received. This procedure permits scoring based on images received by facsimile transmission as well as text received by direct electronic transfer from the testing site.
Once in text form, the written response is compressed, possibly using known secure encryption procedures, and then placed into a file similar to the files used to store bit-mapped images. A subset of the file is sent to the remote scorer using either standard telephone lines or other high speed alternatives. Using this method, then, the ImageBase is composed of compressed/encrypted text in lieu of bit-mapped images of handwriting, so that the size of the ImageBase needed for each scorer""s batch is minimized and can easily be transferred using conventional analog telephone lines and conventional modems.
(a) Individual Remote Scoring with Large Image Files
For projects with few time constraints, but large files of bitmapped images, traditional telephone systems can be utilized if each scorer is provided with a large set of images that the scorer might need to score. The present invention therefore provides apparatus and methods to transfer large sets of responses to individual scorers using recordable CD-ROMS or other high capacity removable random access storage media. In this setting, each scorer is sent a CDROM with all the images that reader might need to view during any of the activities (training, qualifying, or scoring). Once the scorer inserts the appropriate storage media in the appropriate drive, such as a CD-ROM, all images are read directly from the storage media, thereby removing the need to receive transfer of images during processing.
(b) Security of Materials
In addition to the considerations of materials distribution and control of the scoring process, the present invention also provides apparatus and methods to control and secure all data and file access for scorers in all the settings, with special emphasis on data and file transfers for telecommunications using public telephone lines and modems in the various settings that are covered.
The present invention, therefore, provides apparatus and methods to connect users to a network while ensuring absolute security for the operation of the network and of the data on the network.
(i) Network Settings
For users on a LAN, diskless computers and tightly controlled access to programs and data files ensure that scorers have access only to files that they need and that scorers have no ability to view, copy, or modify any files except through the normal activities as described above.
(ii) Individual Remote user Settings
Traditional modem applications, either file transfer programs with or without bulletin boards or remote network connections, have inherent deficiencies for controlling remote scoring, and therefore the present invention utilizes novel apparatus to ensure security of materials.
Even with appropriate scripts, file transfer programs and bulletin boards require more expertise than many scorers can be expected to have, and there is the opportunity for scorers to make errors in the uploading and downloading of files. Furthermore, extensive procedures need to be implemented to ensure that each scorer has access only to his or her files, and to ensure that a sophisticated computer user could not modify a script to gain access to files that should remain secure.
Remote network connections involve security problems that arise when users control the execution of programs on the network, and extensive security precautions would be required to prevent sophisticated computer users from gaining access to files and programs that should be secure.
For individual remote users, then, in accordance with the present invention, all electronic communications are directly controlled by a program running on a node of the central network and acting as both a firewall against unauthorized access and as a manager of all communications to authorized scorers. This program handles all components of file access and control for all remote individual users; it ensures that only valid scorers are connected, and the program itself performs all file access to files on the network so that there can be no unauthorized access to project files. By using encrypted compressed text files when text is used in lieu of images, the system further ensures that no identifying information is transmitted in standard ASCII or text format.